<!DOCTYPE html>
<html lang="en">
    <head>
  <meta charset="utf-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta name="format-detection" content="telephone=no" />

  <title>
    47|案例篇：服务器总是时不时丢包，我该怎么办？（上） | 迪克猪的博客
  </title>

  
  <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png" />
  <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
  <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
  <link rel="manifest" href="/manifest.json" />
  <meta name="theme-color" content="#ffffff" />

  
  <link
    rel="stylesheet"
    href="https://unpkg.com/modern-normalize@0.6.0/modern-normalize.css"
  />

  
  
  
  
  <link rel="stylesheet" href="https://zsy619.github.io/style.min.f7761d111b74dd5c07f0111decee92938c12abc42e0fd319e1a07483e248b54e.css" integrity="sha256-93YdERt03VwH8BEd7O6Sk4wSq8QuD9MZ4aB0g&#43;JItU4=" />

  
  
    
  
</head>

    <body>
        <header id="header">
  <div class="header_container">
    <h1 class="sitetitle">
      <a href="https://zsy619.github.io" title="迪克猪的博客">迪克猪的博客</a>
    </h1>
    <nav class="navbar">
      <ul>
        <li><a href="https://zsy619.github.io">Home</a></li>
        
          <li>
            <a href="/post/">
              
              <span>LINUX性能优化</span>
            </a>
          </li>
        
          <li>
            <a href="/csapp/">
              
              <span>深入理解计算机系统_第三版</span>
            </a>
          </li>
        
          <li>
            <a href="/golang/">
              
              <span>golang</span>
            </a>
          </li>
        
          <li>
            <a href="/docker/">
              
              <span>容器</span>
            </a>
          </li>
        
          <li>
            <a href="/flutter/">
              
              <span>Flutter</span>
            </a>
          </li>
        
          <li>
            <a href="/know/">
              
              <span>知识点</span>
            </a>
          </li>
        
          <li>
            <a href="/categories/">
              
              <span>目录</span>
            </a>
          </li>
        
          <li>
            <a href="/about/">
              
              <span>关于</span>
            </a>
          </li>
        
        <li class="hide-sm"><a href="https://zsy619.github.io/index.xml" type="application/rss+xml">RSS</a></li>
      </ul>
    </nav>
  </div>
</header>

        
<section id="main">
  <article class="post content">
    <h2 class="title">47|案例篇：服务器总是时不时丢包，我该怎么办？（上）</h2>
    <div class="post_content">
      <p>执行 docker ps 命令，查询容器的状态：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                NAMES
dae0202cc27e        feisky/nginx:drop   <span style="color:#e6db74">&#34;/start.sh&#34;</span>         <span style="color:#ae81ff">4</span> minutes ago       Up <span style="color:#ae81ff">4</span> minutes        0.0.0.0:80-&gt;80/tcp   nginx

</code></pre></div><p>执行下面的 hping3 命令，进一步验证 Nginx 是不是真的可以正常访问了。注意，这里我没有使用 ping，是因为 ping 基于 ICMP 协议，而 Nginx 使用的是 TCP 协议。</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
<span style="color:#75715e"># -c表示发送10个请求，-S表示使用TCP SYN，-p指定端口为80</span>
$ hping3 -c <span style="color:#ae81ff">10</span> -S -p <span style="color:#ae81ff">80</span> 192.168.0.30
HPING 192.168.0.30 <span style="color:#f92672">(</span>eth0 192.168.0.30<span style="color:#f92672">)</span>: S set, <span style="color:#ae81ff">40</span> headers + <span style="color:#ae81ff">0</span> data bytes
len<span style="color:#f92672">=</span><span style="color:#ae81ff">44</span> ip<span style="color:#f92672">=</span>192.168.0.30 ttl<span style="color:#f92672">=</span><span style="color:#ae81ff">63</span> DF id<span style="color:#f92672">=</span><span style="color:#ae81ff">0</span> sport<span style="color:#f92672">=</span><span style="color:#ae81ff">80</span> flags<span style="color:#f92672">=</span>SA seq<span style="color:#f92672">=</span><span style="color:#ae81ff">3</span> win<span style="color:#f92672">=</span><span style="color:#ae81ff">5120</span> rtt<span style="color:#f92672">=</span>7.5 ms
len<span style="color:#f92672">=</span><span style="color:#ae81ff">44</span> ip<span style="color:#f92672">=</span>192.168.0.30 ttl<span style="color:#f92672">=</span><span style="color:#ae81ff">63</span> DF id<span style="color:#f92672">=</span><span style="color:#ae81ff">0</span> sport<span style="color:#f92672">=</span><span style="color:#ae81ff">80</span> flags<span style="color:#f92672">=</span>SA seq<span style="color:#f92672">=</span><span style="color:#ae81ff">4</span> win<span style="color:#f92672">=</span><span style="color:#ae81ff">5120</span> rtt<span style="color:#f92672">=</span>7.4 ms
len<span style="color:#f92672">=</span><span style="color:#ae81ff">44</span> ip<span style="color:#f92672">=</span>192.168.0.30 ttl<span style="color:#f92672">=</span><span style="color:#ae81ff">63</span> DF id<span style="color:#f92672">=</span><span style="color:#ae81ff">0</span> sport<span style="color:#f92672">=</span><span style="color:#ae81ff">80</span> flags<span style="color:#f92672">=</span>SA seq<span style="color:#f92672">=</span><span style="color:#ae81ff">5</span> win<span style="color:#f92672">=</span><span style="color:#ae81ff">5120</span> rtt<span style="color:#f92672">=</span>3.3 ms
len<span style="color:#f92672">=</span><span style="color:#ae81ff">44</span> ip<span style="color:#f92672">=</span>192.168.0.30 ttl<span style="color:#f92672">=</span><span style="color:#ae81ff">63</span> DF id<span style="color:#f92672">=</span><span style="color:#ae81ff">0</span> sport<span style="color:#f92672">=</span><span style="color:#ae81ff">80</span> flags<span style="color:#f92672">=</span>SA seq<span style="color:#f92672">=</span><span style="color:#ae81ff">7</span> win<span style="color:#f92672">=</span><span style="color:#ae81ff">5120</span> rtt<span style="color:#f92672">=</span>3.0 ms
len<span style="color:#f92672">=</span><span style="color:#ae81ff">44</span> ip<span style="color:#f92672">=</span>192.168.0.30 ttl<span style="color:#f92672">=</span><span style="color:#ae81ff">63</span> DF id<span style="color:#f92672">=</span><span style="color:#ae81ff">0</span> sport<span style="color:#f92672">=</span><span style="color:#ae81ff">80</span> flags<span style="color:#f92672">=</span>SA seq<span style="color:#f92672">=</span><span style="color:#ae81ff">6</span> win<span style="color:#f92672">=</span><span style="color:#ae81ff">5120</span> rtt<span style="color:#f92672">=</span>3027.2 ms

--- 192.168.0.30 hping statistic ---
<span style="color:#ae81ff">10</span> packets transmitted, <span style="color:#ae81ff">5</span> packets received, 50% packet loss
round-trip min/avg/max <span style="color:#f92672">=</span> 3.0/609.7/3027.2 ms

</code></pre></div><p>从 hping3 的输出中，我们可以发现，发送了 10 个请求包，却只收到了 5 个回复，50% 的包都丢了。再观察每个请求的 RTT 可以发现，RTT 也有非常大的波动变化，小的时候只有 3ms，而大的时候则有 3s。根据这些输出，我们基本能判断，已经发生了丢包现象。可以猜测，3s 的 RTT ，很可能是因为丢包后重传导致的。那到底是哪里发生了丢包呢？</p>
<p><img src="../../images/20200709-1543-39.png" alt=""></p>
<p>从图中你可以看出，可能发生丢包的位置，实际上贯穿了整个网络协议栈。换句话说，全程都有丢包的可能。比如我们从下往上看：</p>
<ul>
<li>在两台 VM 连接之间，可能会发生传输失败的错误，比如网络拥塞、线路错误等；</li>
<li>在网卡收包后，环形缓冲区可能会因为溢出而丢包；</li>
<li>在链路层，可能会因为网络帧校验失败、QoS 等而丢包；</li>
<li>在 IP 层，可能会因为路由失败、组包大小超过 MTU 等而丢包；</li>
<li>在传输层，可能会因为端口未监听、资源占用超过内核限制等而丢包；</li>
<li>在套接字层，可能会因为套接字缓冲区溢出而丢包；</li>
<li>在应用层，可能会因为应用程序异常而丢包；</li>
<li>此外，如果配置了 iptables 规则，这些网络包也可能因为 iptables 过滤规则而丢包。</li>
</ul>
<p>执行下面的命令，进入容器的终端中：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
$ docker exec -it nginx bash
root@nginx:/#

</code></pre></div><h3 id="链路层">链路层</h3>
<p>首先，来看最底下的链路层。当缓冲区溢出等原因导致网卡丢包时，Linux 会在网卡收发数据的统计信息中，记录下收发错误的次数。</p>
<p>可以通过 ethtool 或者 netstat ，来查看网卡的丢包记录。比如，可以在容器中执行下面的命令，查看丢包情况：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
root@nginx:/# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       <span style="color:#ae81ff">100</span>       <span style="color:#ae81ff">31</span>      <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span> <span style="color:#ae81ff">0</span>             <span style="color:#ae81ff">8</span>      <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span> BMRU
lo       <span style="color:#ae81ff">65536</span>        <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span> <span style="color:#ae81ff">0</span>             <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span>      <span style="color:#ae81ff">0</span> LRU

</code></pre></div><p>输出中的 RX-OK、RX-ERR、RX-DRP、RX-OVR ，分别表示接收时的总包数、总错误数、进入 Ring Buffer 后因其他原因（如内存不足）导致的丢包数以及 Ring Buffer 溢出导致的丢包数。
TX-OK、TX-ERR、TX-DRP、TX-OVR 也代表类似的含义，只不过是指发送时对应的各个指标。</p>
<p>注意，由于 Docker 容器的虚拟网卡，实际上是一对 veth pair，一端接入容器中用作 eth0，另一端在主机中接入 docker0 网桥中。veth 驱动并没有实现网络统计的功能，所以使用 ethtool -S 命令，无法得到网卡收发数据的汇总信息。</p>
<p>容器终端中，执行下面的 tc 命令，不过这次注意添加 -s 选项，以输出统计信息：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
root@nginx:/# tc -s qdisc show dev eth0
qdisc netem 800d: root refcnt <span style="color:#ae81ff">2</span> limit <span style="color:#ae81ff">1000</span> loss 30%
 Sent <span style="color:#ae81ff">432</span> bytes <span style="color:#ae81ff">8</span> pkt <span style="color:#f92672">(</span>dropped 4, overlimits <span style="color:#ae81ff">0</span> requeues 0<span style="color:#f92672">)</span>
 backlog 0b 0p requeues <span style="color:#ae81ff">0</span>

</code></pre></div><h3 id="网络层和传输层">网络层和传输层</h3>
<p>在网络层和传输层中，引发丢包的因素非常多。不过，其实想确认是否丢包，是非常简单的事，因为 Linux 已经为我们提供了各个协议的收发汇总情况。</p>
<p>执行下面的 netstat -s 命令，就可以看到协议的收发汇总，以及错误信息了：</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">
root@nginx:/# netstat -s
Ip:
    Forwarding: <span style="color:#ae81ff">1</span>          //开启转发
    <span style="color:#ae81ff">31</span> total packets received    //总收包数
    <span style="color:#ae81ff">0</span> forwarded            //转发包数
    <span style="color:#ae81ff">0</span> incoming packets discarded  //接收丢包数
    <span style="color:#ae81ff">25</span> incoming packets delivered  //接收的数据包数
    <span style="color:#ae81ff">15</span> requests sent out      //发出的数据包数
Icmp:
    <span style="color:#ae81ff">0</span> ICMP messages received    //收到的ICMP包数
    <span style="color:#ae81ff">0</span> input ICMP message failed    //收到ICMP失败数
    ICMP input histogram:
    <span style="color:#ae81ff">0</span> ICMP messages sent      //ICMP发送数
    <span style="color:#ae81ff">0</span> ICMP messages failed      //ICMP失败数
    ICMP output histogram:
Tcp:
    <span style="color:#ae81ff">0</span> active connection openings  //主动连接数
    <span style="color:#ae81ff">0</span> passive connection openings  //被动连接数
    <span style="color:#ae81ff">11</span> failed connection attempts  //失败连接尝试数
    <span style="color:#ae81ff">0</span> connection resets received  //接收的连接重置数
    <span style="color:#ae81ff">0</span> connections established    //建立连接数
    <span style="color:#ae81ff">25</span> segments received      //已接收报文数
    <span style="color:#ae81ff">21</span> segments sent out      //已发送报文数
    <span style="color:#ae81ff">4</span> segments retransmitted    //重传报文数
    <span style="color:#ae81ff">0</span> bad segments received      //错误报文数
    <span style="color:#ae81ff">0</span> resets sent          //发出的连接重置数
Udp:
    <span style="color:#ae81ff">0</span> packets received
    ...
TcpExt:
    <span style="color:#ae81ff">11</span> resets received <span style="color:#66d9ef">for</span> embryonic SYN_RECV sockets  //半连接重置数
    <span style="color:#ae81ff">0</span> packet headers predicted
    TCPTimeouts: <span style="color:#ae81ff">7</span>    //超时数
    TCPSynRetrans: <span style="color:#ae81ff">4</span>  //SYN重传数
  ...

</code></pre></div><p>netstat 汇总了 IP、ICMP、TCP、UDP 等各种协议的收发统计信息。不过，我们的目的是排查丢包问题，所以这里主要观察的是错误数、丢包数以及重传数。</p>
<p>根据上面的输出，你可以看到，只有 TCP 协议发生了丢包和重传，分别是：</p>
<ul>
<li>11 次连接失败重试（11 failed connection attempts）</li>
<li>4 次重传（4 segments retransmitted）</li>
<li>11 次半连接重置（11 resets received for embryonic SYN_RECV sockets）</li>
<li>4 次 SYN 重传（TCPSynRetrans）</li>
<li>7 次超时（TCPTimeouts）</li>
</ul>
<p>这个结果告诉我们，TCP 协议有多次超时和失败重试，并且主要错误是半连接重置。换句话说，主要的失败，都是三次握手失败。</p>
<h3 id="小结">小结</h3>
<p>网络丢包，通常会带来严重的性能下降，特别是对 TCP 来说，丢包通常意味着网络拥塞和重传，进一步还会导致网络延迟增大、吞吐降低。</p>

    </div>
    <div class="info post_meta">
      <time datetime=2020-07-09T15:34:15&#43;0800 class="date">Thursday, July 9, 2020</time>
      
        <ul class="tags">
        
          <li> <a href="https://zsy619.github.io/tags/linux%E6%80%A7%E8%83%BD%E4%BC%98%E5%8C%96">linux性能优化</a> </li>
        
          <li> <a href="https://zsy619.github.io/tags/linux">linux</a> </li>
        
        </ul>
      
      
    </div>
    <div class="clearfix"></div>
  </article>
  
    <div class="other_posts">
      
      <a href="https://zsy619.github.io/post/46%E6%A1%88%E4%BE%8B%E7%AF%87%E4%B8%BA%E4%BB%80%E4%B9%88%E5%BA%94%E7%94%A8%E5%AE%B9%E5%99%A8%E5%8C%96%E5%90%8E%E5%90%AF%E5%8A%A8%E6%85%A2%E4%BA%86%E5%BE%88%E5%A4%9A/" class="prev">46|案例篇：为什么应用容器化后，启动慢了很多？</a>
      
      
      <a href="https://zsy619.github.io/post/48%E6%A1%88%E4%BE%8B%E7%AF%87%E6%9C%8D%E5%8A%A1%E5%99%A8%E6%80%BB%E6%98%AF%E6%97%B6%E4%B8%8D%E6%97%B6%E4%B8%A2%E5%8C%85%E6%88%91%E8%AF%A5%E6%80%8E%E4%B9%88%E5%8A%9E%E4%B8%8B/" class="next">48|案例篇：服务器总是时不时丢包，我该怎么办？（下）</a>
      
    </div>
    <aside id="comments">
</aside>

  
</section>

        <a id="back_to_top" title="Go To Top" href="#">
  <span>
    <svg viewBox="0 0 24 24">
      <path fill="none" d="M0 0h24v24H0z"></path>
      <path d="M12 2L4.5 20.29l.71.71L12 18l6.79 3 .71-.71z"></path>
    </svg>
  </span>
</a>

        <footer id="footer">
  <p>
    <span>&copy; 2021 <a href="https://zsy619.github.io" title="迪克猪的博客">迪克猪的博客</a> </span>
    <span>Built with <a rel="nofollow" target="_blank" href="https://gohugo.io">Hugo</a></span>
    <span>Theme by <a rel="nofollow" target="_blank" href="https://github.com/wayjam/hugo-theme-mixedpaper">WayJam</a></span>
  </p>

  <script src="https://zsy619.github.io/js/main.min.8b182175f5874aeed0acc0979345c98d4bde22208ec4f36cc1d6e3102acb4b10.js" integrity="sha256-ixghdfWHSu7QrMCXk0XJjUveIiCOxPNswdbjECrLSxA=" crossorigin="anonymous" async></script>
</footer>

    </body>
</html>
